DOCUMENT RESUME 



ED 356 306 UD 029 132 

AUTHOR Stevens, Floraline I.; Grymes, John 

TITLE Opportunity To Learn: Issues of Equity for Poor and 

Minority Students. 
INSTITUTION National Center for Education Statistics (ED), 

Washington, DC. 
REPORT NO NCES-93-232 
PUB DATE Jan 93 

NOTE 69p. 

PUB TYPE Reports - Research/Technical (143) 

Tests/Evaluat ion Instruments (160) 

EDRS PRICE MF01/PC03 Plus Postage. 

DESCRIPTORS Data Analysis; Disadvantaged Youth; ^'^Economically 

Disadvantaged; ''^Educationally Disadvantaged; 
Elementary Secondary Education; '"^Equal Education; 
High Risk Students; Mail Surveys; ''^Minority Group 
Chi Idren ; Outcomes of Education; Poverty; Publi c 
Schools; ''^'School Districts; School Personnel; School 
Statistics; Urban Schools 

IDENTIFIERS ''^Educational Information; '''Opportunity to Learn 

ABSTRACT 

This study evaluated how public school districts 
collect, disaggregate, and analyze student outcome data. A survey was 
mailed to research directors and test directors responsible for 
assessments in 1A2 public school districts to determine if the 
districts have the capacity to analyze opportunity to learn (OTL) 
data, what their interest level is in the data, and what obstacles 
they face in collecting and analyzing OTL data. At least 20 percent 
of the respondents received follow-up telephone interviews. District 
size, urbanicity, and geographic area vere analyzed using open-ended 
responses. Ninety-one (64 percent) of the school districts responded. 
Respondents ranged from program evalnators to assistant 
superintendents; 56 percent were directors and managers of research 
and evaluation, and 61 percent came from urban public school 
districts. Results show that OTL is virtually an unknown concept in 
the United States. Students' differences in academic achievement are 
not being related to an analysis of OTL. Most districts limit test 
data disaggregation to race/ethnicity. Lack of OTL information 
hampers teachers' abilities to improve their teaching practices. 
Policy recommendations include: (1) raise the awareness and knowledge 
levels of all parties responsible for educating students; and (2) 
train public school district personnel to use OTL information, 
encourage OTL data collection, and do more research on the quality of 
instructional delivery. Fourteen data tables, the OTL Questionnaire, 
"Opportunity To Learn: The Need for a Comprehensive Definition for 
Substantive Data Analysis" (Floraline I. Stevens), and the OTL Survey 
Follow-Up Interview Protocol are included. (Contains 49 references.) 
(RLC) 

ic:iirfci(icic:f(i(i(ici*c'k^'ki<fc'if)ii<->\'k-k^*^->'c Vc >'c * it:k-k-k Vc Vc -k i: -fc ic -k -k -k k-kickick k k icitkit k k k k kkifkt'c k k ic k:kk 

''^ Reproductions supplied by EDRS are the best that can be made * 

from the original document. ''^ 

i<ici(kickirk'fck'kk^'ckki(-kkkkin\k'kkic-kk-kif'kixin\kiiixi<^ 



Opportunity 




Issues of Equity 
for Poor and Minority 

Students 



jS OEPAWTMENT OF^ EDuCAnON 

M f "a'^gr* ".I*** r^J"^ ' .txpf.-.o 
. 



ST COPY AVAILABLE 



z 



Opportunity 

TO 

LearN 

Issues of Equity 
for Poor and Minority 

Students 

Floraline L Stevens 
Los Angeles Unified School District 

assisted by 



John Grymes 
National Center for Education Statistics 



NATIONAL CENTER FOR EDUCATION STATSSTICS 



U.S. Department of Education 

Lamar Alexander 
Secretaiy 

Office of Educational Research and Improvement 

Diane Ravitch 
Assistant Secretary' 

National Center for Education Statistics 

Emerson J. Elliott 
Commissioner 



January 1993 



This work was performed while the author was the 
American Educational Research Association's Senior 
Research Fellow at the National Center for Education 
Statistics (NOES). The results, interpretations, and 
recommendations in this paper are those of the author and 
do not reflect findings, recommendations, or policies of 
NCES. 



Acknowledgments 



This paper was prepared as part of my fellow's research project at the National Center for 
Education Statistics (NCES) from August 1991 through June 1992. The funding came from 
the American Educational Research Association (AERA). I especially appreciate the 
help and assistance given to me by the staff at AERA, particularly, Bill Russell, Executive 
Director, Jerry Sroufe, Jaime Hitchcock, and Linda Goff. 

Special thanks to the members of the AERA Grant Advisory Board and especially its 
chair, Richard Shavelson, for their vision and support of a fellows program at NCES. 

Heartfelt thanks to the 99 public school directors who responded to the survey. Their 
cooperation was inuispensable. 

Emerson Elliott, Commissioner of Education Statistics, provided me with encouragement 
and insightful advice throughout the project. Many NCES stuff members provided advice 
and reviewed earlier versions of this paper, including Associate Commissioners Jeanne 
Griffith (Fellows Liaison), Shirley Jackson, and Paul Planchon; Sharon Bobbitt, Salvatore 
Corrallo, Lee Hoffman, Dan Kasprzyk, Jeff Owings, Lois Peak, John Ralph, and Mary 
Rollefson. Special thanks to Lois Peak for sharing with me many articles on international 
research. 

John Grymes of NCES was indispensable to this project. His professional help and 
advice, as well as his computer analysis of the survey data, made it possible for me to 
write the results for this paper in a timely manner. He was a dedicated assistant during 
this entire period. Special thanks also to Mark Root at the Council of the Great City 
Schools for producing the charts in this report. 

The opportunity to be involved in NCES activities provided me with information that 
assisted me in organizing the contents of this paper. The acceptance of my role as a 
research fellow by the NCES staff made this experience most positive. 



Floraline L Stevens 

Los Angeles School District 

Senior Research Fellow 



HI 



Table of Contents 



Acknowledgments > 

The Background: Raising the Issue 1 

Understanding Opportunity to Learn 5 

The Study: What's Happening with Opportunity to Learn in the Public Schools 15 

What Do These Findings Tell Us 31 

What Are the Policy Recommendations 37 

Conclusion 39 

Appendix A 

Appendix B 

Appendix C 

Appendix D 



ERLC 



The Background: Raising the Issue 



Opportunity to learn the designated curriculum for a grade level or age group is a major 
equity issue for students who are at risk of not developing academically to their fullest 
potential. As a group, at-risk students are often categorized as poor, disadvantaged, or 
from the underclass. Past and current studies of at-risk students, most of whom are in 
public schools, have centered on the demographics of poverty and minority membersnip 
(CPRE, 1991; Orland, 1990; Oakes, 1990; and Coleman, 1966). Whatever terms used to 
describe these students, the principal issue is that we are not currently meeting the 
educational needs of these students. If this is due to explicit or implicit policies that make 
portions of our population expendable, then we are not meeting the needs of the nation 
economically, socially, or morally. 

The coming technological and information-based era requires more skilled and talented 
persons to emerge from our schools. Policymakers, teacher educators, school 
administrators, and particularly classroom teachers must consider the issue of equity in 
preparing these students for the world. Equally important is whether students are being 
given the opportunity to learn what is needed. 

The Special Study Panel on Education Indicators in Education Counts (1991) focused on 
the issue of opportunity to learn when they said, "Tlie concern for educational equity is 
based on a fundamental belief in fairness. It transcends political boundaries and the 
narrow issues of interest groups. In American society, the values of fairness and justice 
are deeply held..." and, "We need to know if students at risk have access to the full range 
of educational opportunities, what kinds of learning opportunities are provided, and how 
well-tailored they are to the educational needs of these students." 



At the State Level: Reforms with Poor Results 

The school reform movement in the United States called for drastic changes in public 
school education to meet the needs of at-risk students and the nation. Many solutions 
were offered and tried. On the national level, education reforms included new national 
goals. New American Schools, world class standards, a national curriculum, and 
measurements of the nation's educational health. 

States have tried graduation course requirements, accountability indicators, student 
competency testing, teacher competency testing, teacher certification, curriculum revisions, 
and adoption of new curriculum frameworks. Many states expended tremendous sums of 
money to implement their reform solutions. Unfortunately, despite all the reform efforts, 
academic achievement continues to be low for poor and minority students in most large, 
urban public school districts. Obviously, something is missing. 
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Why? One School District's Experience 

In the mid-1980s, the board of education in a large urban school district in California 
responded to reports of low academic achievement among its students, most of whom 
were poor and minority, with two major programs. One was a major bilingual education 
program, and the other was a special program in ten elementary schools with African 
American student populations of 60 percent or more. This special program's evaluation 
included assessing student progress in improving academic achievement in reading, 
mathematics, and English language arts over a 5-year period. 

Assessment of grade-level, subject-matter skills/objectives would determine progress 
toward meeting the overall goal. The teachers in the ten schools decided to administer 
interval or segmented tests (clusters of items for a few skills for each subject area) every 8 
weeks. These tests provided information to the teacher about their students' mastery of 
grade-level skills/objectives prior to the district's end-of-year, norm-referenced testing 
program. 

When the intei-val test information was collected and analyzed, the results for some 
classrooms showed that over 80 percent of the students had not mastered some of the 
skills/objectives for the interval. The testing unit queried some of the teachers about 
whether or not the subject matter skills/objectives were taught. Most of the teachers 
responded that they covered the skills/objectives in their classrooms. However, in 
subsequent grade level meetings, these teachers asked questions about what were 
appropriate or alternative strategies for teaching these skills. Their questions suggested 
that some teachers had not taught the skills, or if they had, their instructional delivery was 
not effective enough for the students in their classrooms to master the subject matter. 



The Teacher Is Critical to Student Academic Achievement 

At this point, questions arose about whether the teachers at these and other schools in the 
district understood fully what to do with reports of student academic achievement scores 
when the scores were not disaggregated a:id there was no accompanying information 
provided about student exposure to, and coverage of, the subject matter. Program staff 
and teachers realized that students who were not exposed to grade-level, subject-matter 
skills and concepts would not score as "average" on the district's grade-level, 
norm-referenced tests. Instead, the test data would report low academic achievement with 
the incorrect assumption that the students did not work hard enough or were not capable 
of learning the subject matter, when in fact, student performance was tied critically to the 
performance of their classroom teachers. 

Although originally designed to monitor students' academic progress, the interval tests 
provided interesting information about students' access to the curriculum and raised 
questions about the quality of instructional delivery. 
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The result was the emergence of the issue of opportunity to learn. Because the teacher's 
role is important in determining opportunity to learn, this paper focuses on the teacher's 
role in this paradigm (Bloom, 1976; Goodlad, 1983; Oakes, 1983). Administrators need to 
help teachers and other support staff jn understanding the issue's impact on instructional 
equity and to assist teachers in implementing instructional models and programs that will 
promote access to learning for poor and minority students (Muiphy, 1988). 

The Importance of External Support 

Many poor and minority students do not live in environments that provide support to 
make up for any deficiencies that might exist in the school. In contrast, students from 
most middle and upper class homes have supportive, literate, and learning environments 
to supplement and enrich school instruction (Adams, 1990). Therefore, quality of 
instruction is of paramount importance for poor and minority students or any other 
students from inadequate backgrounds. 

As Adams explained, students who arrive at school behind in their prereading skills will 
not test as "average" or above, will be judged as slow learners or educationally 
handicapped, and will be treated accordingly. Based on this information, it is apparent 
that without intervention, the race could be lost for these students before it has even 
begun. From the preschool level on, opportunity to learn has significant importance for 
poor and minority students. 

Creating a "Lower Class" 

In an article in Phi Delta Kappan (1991), Bracey noted that there are menial jobs in our 
country that must be performed and that over-education could generate social problems 
because well-educated people will not want to do the menial work. Writing that some 
people believe "we must continue to produce an uneducated social class" to do this menial 
work, Bracey acknowledged the equity or equal opportunity issue in this viewpoint. 

Researchers, policymakers, practitioners, and the public-at-large need to question the 
morality of those who hold this viewpoint. Who decides who will be relegated to this role? 
In reality, no group willingly or knowingly would choose to be at the bottom of the 
socioeconomic ladder. However, by not acting on the issue of opportunity to learn in our 
public schools, we are actively relegating a group of students to the "lower class." 

Kirst (1991) wrote, "The low levels of education found in big cities and among minorities 
are appalling and are a factor in America's weak showing in international comparisons of 
least-skilled workers." Why is education in big cities failing minority and poor students? 
Past research blamed the poor academic performance of these students on their poverty 
and minority status (Coleman, et al., 1966). This popular and misguided belief assumed 
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that schools could not change these students' academic performances. However, 
subsequent research shows that academic performance, no matter the students' 
backgrounds, can change with an im.provement in the quality of instruction. Specifically, 
changes in how time is spent in the classrooms, how learning is organized, what 
curriculum materials are used, what attitudes are reinforced, what beliefs and values arc 
operant, and how supportive the conditions for teaching and learning are will make a 
difference in achievement (Sirotnik, 1990; Brophy and Good, 1986). 



Time to Consider New Variables 

Under the circumstances, we must seriously question the accuracy of reports about student 
academic achievement. When we consider student outcome data (test scores, etc.) in 
relation to their opportunity to learn, we realize a different kind of inquiry is needed 
(Goodlad, 1983; Goodlad and Oakes, 1988; Murphy, 1988; Sirotnik, 1990). Investigation of 
student outcomes should include these questions; 
I 

What is opportunity to learn? 

2. What is the capacity of the public school districts to collect and use 
opportunity to learn information in their analyses of student outcome 
data? 

3. How interested are public school districts in collecting and using 
opportunity to learn data in analyzing student outcome data? 



Understanding Opportunity- to Learn 



Tlie powerlii! concept of opportunity to learn has been used principally to explain 
differences mong students in comparative international r,tudies of educational 
achievement and in some small-scale national research studies. Variables gleaned from 
these studies focus on what teachers do in their classrooms when they are teaching 
students. The variables are: (1) content coverage; (2) content exposure; (3) content 
emphasis; and (4) quality of instructional delivery (Table 1). With these variables to build 
a conceptual framework, those responsible for analyzing and reporting student outcome 
data could investigate more substantively the reasons for the differences they perceived in 
academic achievement. The framework also provides a basis for meaningful 
recommendations to improve teaching and to meet students' educational needs (Table 2). 

According to the studies: 

* Content coverage investigates whether or not students covered the core curriculum 
for a particular grade level or subject area (for example, grade 4 reading or 
algebra). 

* Content exposure questions the time allowed and the depth of teaching (time-on- 
task). 

* Content emphasis determines which topics within the curriculum are selected for 
emphasis and which students are selected to receive low or higher order skills. 

* Quality of instructional delivery reveals how the teaching practices in the 
cla.ssroom impact student academic achievement (coherent presentation of lessons). 

Earlier studies based their definition of opportunity to learn principally on how the 
variables were analyzed. They focused on the implemented curriculum by examining 
content coverage (the overlap of curriculum content and test content), teacher 
decisionmaking regarding content emphasis, and content exposure. Extensive research on 
the effects of instructional practices was not connected with the opportunity to learn 
research. Quality of instructional delivery was viewed as a separate equity issue. 

A Look at the National Studies 

1. Content Coverage: Curriculum Content and Test Content Overlap 

Overlap research showed that in many instances curriculum content and test content did 
not match (Leinhardt, 1983; Leinhardt and Seewald, 1981). These studies confirmed an 
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Table 1. Opportunity to learn definitions gleaned from research studies 



Author ^>ate of 

Study 

1. Walker and 1974 
Schaffarzick 

2. LeMahieu and 1?79 
Leinhardt 

3. Leinhardt and 1981 
Seewald 

Leinhardt 1983 

1987 

5. Yoon, et al. 1990 

6. Wiley 1990 

7. Schmidt 1990 

8. Shavelson and Stern 1981 
Floden, et al. 1981 



Definition^ 

Content coverage: how many of the items 
on the test match the curriculum that was 
taught 

Content emphasis: high student test scores 
influence content coverage. The content 
of the curriculum is controlled by the test. 

Content coverage: how many of the items 
on the test match the curriculum that was 
taught. 

Content coverage: adequate and timely 
instruction of specific content and skills 
prior to taking the test. 

Content exposure: time spent to cover the 
specific content. 

Content coverage: when a topic is aligned 
with a particular course, teachers will 
validly report if the topic was taught. 

Content exposure: the more time one 
spends on instructional experiences, the 
more one learns or time-on-task. 

Content coverage: individual test items 
determine small units of content covered. 

Content emphasis: textbooks and 
materials prepared by teachers were most 
influential on selection of content to teach 
Next, teachers were influenced by course 
syllabi. 

Content empha<:is: consent is selected by 
teachers based on personal experiences, 
particular proficiency in the topic, topic 
viewed as important, textbooks used, etc. 



Table 1. Opportunity to learn definition gleaned from research studies - (continued) 



Author Date of 

Study 

9. Goldenberg and 1991 
Galiimore 



10. McDonnell, et 1990 
al. 



11. Brophy and Good 1986 



Definition* 

Content emphasis: teachers' 
attitudes lower the learning ceiling for 
some students which reduces the level 
and quantity of work students are 
expected to complete in the classroom. 

Content emphasis: curriculum 
offerings differentiated according to 
student ability levels. Teachers place 
different amounts of emphasis on 
different objectives. 

Content exposure: academic learning is 
influenced by the amount of time that 
students spend engaged in appropriate 
academic tasks. 



12. Stevenson and 1992 
Stigler, et al. 



Quality of instructional delivery: 
teachers first structure new information 
for students, help them relate it to what 
they already know, monitor their 
performance, and provide corrective 
feedback during the lessons. This applies 
to any body of knowledge or set of skills. 
Teachers did not release their students to 
begin work until it had been explained 
thoroughly. 

Quality of instructional delivery: a good 
lesson is coherent. It is highly organized 
having an introduction, a conclusion, and 
a content theme. The teacher r needed 
to relate the different parts of a lesson to 
one another, to explain the 
interrelatedness of the various activities, 
or coherence is lost. 



*Note: The definitions of opportunity to learn are based principally on how 
the data are analyzed. 



Table 2, Opportunity to Learn: A conceptual framework developed from 

international and national research studies to investigate students' access to 
the core curriculum. 



Variable 
1. Content Coverage 

(V^alker and Schaffarzick, 1974; 
Leinhardt and Seewald, 1981; 
Leinhardt, 1983; Winfield, 1987; 
Yoon, et al., 1990; Wiley, 1990) 



2. Content Exposure 

(Winfield, 1987; Wiley, 1990; 
Brophy and Good, 1986) 



3. Content Emphasis 

(LeMahieu and Leinhardt, 1979; 
Floden, et al, 1981; Shavelson and 
Stern, 1981; Oakes, 1990; Goldenberg 
and Gallimore, 1991; McDonnell et 
al., 1990) 

4. Quality of Instructional Delivery 

(Stevenson and Stigler, 1992; 
Brophy and Good, 1986) 



Definition 

Teacher arranges for all students to have 
access to the core curriculum. Teacher 
arranges for all students to have access 
to critical subject matter topics. Teacher 
ensures there is curriculum content and 
test content overlap. 

Teacher organizes class so that there is 
time-on-task for students. Teacher 
provides enough time for students to 
learn the content of the curriculum and 
to cover adequately a specific topic or 
subject. 

Teacher selects topics from the 
curriculum to teach. Teacher selects the 
dominant level to teach the curriculum 
(recall, higher order skills). Teacher 
selects which skills to teach and which 
skills to emphasize to which groups of 
students (ability grouping and tracking or 
regrouping). 

Teacher uses teaching practices 
(coherent lessons) to produce students' 
academic achievement. Teacher uses 
varied teaching strategies to meet the 
educational needs of all students. 
Teacher has cognitive command of the 
subject matter. 



important aspect of investigating opportunity to learn-the consideration of curricula and 
test overlap when test scores are used to judge student and program success. 

To determine overlap, Leinhardt asked three questions of teachers with at least 3 to 4 
years experience: 

1. Did you teach that information this year? 

2. Has Student X been taught enough information to answer the item correctly? 

3. Can Student X get this item right? 

Question 2 was asking whether the teacher had taught the material Using a blank test, 
the teacher was asked to go through each item in the test for each child in the class. An 
overlap estimate (percentage of overlap) was determined by the number of items the 
teacher claimed to have taught divided by the number of items on the test times one 
hundred. Leinhardt found that when teachers answered questions about individual 
children, they were more conservative and precise than with hypothetical or general cases. 



2 Content Coverage and Content Exposure 

Winfield (1987) investigated the same question about curriculum content and test content 
when she secured teachers* estimates of test content covered in a first grade Title 1 class 
in reading. Winfield defined opportunity to learn as the provision of adequate and timely 
instruction of specific content and skills prior to taking a test. She suggested that 
opportunity to learn might also be measured by components such as time spent in 
reviewing, practicing, or applying a particular concept or by the amount and depth of 
content covered with particular groups of students. 

To assess coverage and exposure, examples of items similar ti those on the CTBS reading 
test were constructed using the test format and words from the Dolch first grade reading 
test. Actual CTBS items were not used. The survey included 25 items similar to those 
found in the word attack section. Teachers were instructed to respond to each te^t item 
by answering survey questions using a series of five-point response scales (For example: 
None, 1-9 minutes, 10-19 minutes, 20-29 minutes, more than 30 minutes; not at all, yearly, 
monthly, weekly, daily; and not in curriculum, introduced only, introduced and taught 
slightly, introduced and taught for mastery, critical objective). 

Examples of survey questions were: 

1. How much time did you use to introduce this item/concept to Title 1 students? 



2. How confident are you that the majority of your Title 1 students have mastered 
this item/concept? 

3. To what extent is this item/concept emphasized in the school reading curriculum 
for first grade? 



Winfield discovered that the teachers of first grade Title I students emphasized 
word-attack skills more than they did comprehension skills. This finding was contrary to 
research which noted that instruction in decoding should be accompanied by instruction in 
comprehension to foster an ability to read independently (Doyle, 1983). 

3. Content Coverage 

Another overlap study by Walker and Schaffarzick (1974) used multivariate analysis to 
more accurately determine student academic achievement. They used opportunity to learn 
the content tested on a standardized test battery as the covariate in the analysis. They 
warned that failure to consider variation in opportunity to learn the subject matter can 
result in student differences (test scores) that may be mistakenly attributed to program or 
student characteristics when the differences may actually lie in the match or mismatch 
between what is tested and what is taught. Therefore instructional processes in the 
classroom affect overlap further. If the curriculum taught in the classroom matches what 
is tested, most students will answer the test items correctly. 

4. Content Emphasis: Teacher Decisionmaking 

Other studies proposed that opportunity to learn should be investigated in the context of 
teachers as decisionmakers of the content covered. Floden, et al. (1981) and Shavelson 
and Stern (1981) showed that teachers choose what they want to emphasize in their 
classrooms for various reasons. They base their choices on personal experiences, personal 
proficiency in a topic, perception of certain topics as being more important, professional 
ex-periences, past experiences with having to remediate repeatedly certain topics, and 
influence of past professors, education courses, textbooks, and other authorities. 

LeMahieu and Leinhardt (1985) found that students' high test scores influenced content 
coverage. Since most teachers perceive high test scores as good, they strive to increase 
the overlap between instruction and test content. Subsequently, tests have a powerful 
influence on teacher selection of instructional content. The test content controls the 
opportunity to learn a full curriculum. In this respect, LeMahieu and Leinhardt cautioned 
against tests assuming "the status of de facto definition of the domain under investigation." 

5. Content Coverage 

Yoon, et al. (1990) built on previous studies that used teachers' self-reporting content 
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coverage and investigated the validity of teachers* reports of mathematical content 
exposure and coverage. Their study was based on the premise that teachers' reports are 
potentially valuable, efficient, and cost effective. 

Yoon and his associates used seven responses taken from the Second International 
Mathematics Study (lEA, 1982) to collect data on content coverage. Teachers were asked 
to indicate the degree topics that were covered in each mathematics course that they 
taught. The response options were: new, extended, reviewed, assumed, taught later, not in 
the curriculum, and don't know. They found that if a topic was aligned with a course, 
virtually all teachers who claimed to teach the course chose the response options, "new" 
and "extended," and student performance on these test items was consistent with teachers' 
reports of coverage. 

6. Content Emphasis 

Goldenberg and Gallimore (1991) demonstrated in their case study of Spanish-speaking 
first grade students in one school that teachers based their attitudes on demographics and 
background information of the students' parents. The teachers believed that these students 
were not ready for the prescribed first-grade curriculum and were also convinced that the 
kindergarten students were not ready for reading and writing despite evidence to the 
contrary. As a result, the teachers reduced the level of schoolwork assigned to the 
students. They believed that slow and tortuous progress in the reading curriculum was a 
fact of life and that only a few children would be on grade level by the end of the 
first-grade. 

International Studies 

The need to understand differences in student outcome data across nations generated 
survey items on opportunity to learn in international studies, but information was not 
solicited about outcomes for poor and minority students. In these studies, items on 
opportunity to learn inquired principally about curriculum and content coverage, 
curriculum emphases, and time allocated to instruction in the classroom. The Second 
International Mathematics Study (SIMS), sponsored by the International Association for 
the Evaluation of Educational Achievement (lEA), produced a major collection of 
information on opportunity to learn. 

Recent large international studies with items on opportunity to learn included the first and 
second International Educational Assessment of Progress (lEAP) conducted by the 
Educational Testing Service and sponsored by the National Center for Education Statistics 
(NCES). Two additional lEA studies using ite s on opportunity to learn are the Study of 
Reading Literacy and, in its planning stage. The Third International Mathematics and 
Science Study (TIMSS). Tlie Organization for Economic Cooperation and Development 
(OECD) expressed concern and criticism about the opportunity to learn definition used in 
several international reports. OECD (1991) felt that when investigating content coverage 
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the definition of "been taught" should be standardized from country to country and from 
teacher to teacher. Also, it suggested taking steps to reduce differences in teacher 
judgment regarding content coverage. 

In preparation for the TIMSS, Burstein, et al. (1991) reviewed and displayed various items 
on opportunity to learn used in studies of achievement in science and mathematics. Some 
typical items asked of teachers were: 

o What percent of the textbook/workbook would they cover when teaching the 
course? 

o How many class periods did they devote to a particular topic,, e.g., complex 
numbers in algebra? 

o Did all students study the same content in a class? 

o What was the highest level of coverage reached by the students doing 
ma hematical problems in the class, e.g., memorize, routine problems, novel 
problems, develop proofs? 

o What was important in determining what they taught on a day-to-day basis, e.g., 
how important was what the students will need in the next grade or in the next 
course in the subject? 

o Did they assign homework and how often? 

These items reflect a traditional approach to investigating opportunity to learn. They 
collect information on content exposure that measures the amount of exposure of 
particular students to instructional experiences (the more time one spends, the more one 
learns) and content coverage that measures content covered in srnall ^nits such as 
individual test items (Wiley, 1990). 

The Equity Issue in Public Schools 

Opportunity to learn studies did not investigate equity issues in public schools in the 
United States and in other nations. However, the Special Study Panel on Education 
Indicators (1991) identified equity as an important issue needing indicators in its report to 
the NCES. The Panel urged that the nation needed "more sensitive barometers" of 
students' learning opportunities. In response, Bobbitt (1992) pointed out that NCES is 
currently investigating some aspects of educational equity; however, there is room for 
improvement in studying opportunity to learn in the elementary and secondary data 
collection system. 
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To expand, at the public school district level, little is known about how or whether there is 
much concern about addressing opportunity to learn for quality education purposes in 
general and for equity purposes for poor and minority students in particular. Based on 
this lack of information, we conducted a survey. 
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The Study: What's Happening with Opportunity 
to Learn in the Public Schools 



We planned this study to obtain information about how public school districts handle the 
collection and analysis of student outcome data. Our intent was to determine if the 
districts have the capacity to analyze opportunity to learn information, what their interest 
level is in the data, and what obstacles they face in collecting and analyzing opportunity to 
learn information. 



Methods 

We mailed a survey form to a purposive sample of research directors and test directors 
responsible for assessment programs in 142 public school districts. Our purpose was to 
determine the capacity of their districts to disaggregate and analyze student outcome data 
and their interest and use in opportunity to learn data. The survey form was accompanied 
by a paper brief that provided information about opportunity to learn (Appendix A). The 
paper described why there was a need to conduct educational research in this area. 

With our sample, we wanted to reach those responsible for implementing assessment 
policies and for reporting student outcome information for decisionmaking and 
accountability purposes. We also wanted to reach people trained in research, evaluation, 
and assessment. In particular, we wanted to survey as many directors of research and 
evaluation in the largest U.S. urban public school districts as possible. Therefore, we 
selected all directors of research and evaluation who attended the annual AERA meeting, 
the research directors who attended the research liaison meeting of the Council of Great 
City Schools, and all members of the National Association of Test Directors who worked 
iu public schools. With these selections some overlap in the sampling does exit. To avoid 
duplication of responses from the same district, the subordinate member's responses were 
not tallied. 

These questions guided the development of the survey items: 

1. What is the structure/organization of public school districts' testing programs? 

2. What capacity do public school districts have to disaggregate test data by 
demographics and background information and how interested are they in doing it? 

3. What is the interest of public school districts in using opportunity to learn 
variables and in using different testing processes in the analysis of student data? 

4. For public school districts, is the opportunity to learn definition sufficient? 
Should the definition be expanded? 
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Following the receipt of the completed surveys, we drew a subsample of at least 20 
percent of the respondents, and followed up with telephone interviews to obtain more 
indepth information regarding their responses. 

Analysis 

Descriptive statistics give the number and the percentage of responses for each of the 
items. We also analyzed data across such variables as size of the district, urbanicity, and 
geographic area of the district Open-ended responses were summarized into categories. 
Interview comments provided explanatory information. 



Findings 

Ninety-one or 64 percent of the 142 school districts we contacted responded to the 
questionnaire. The respondents' positions ranged from program evaluators to assistant 
superintendents. However, 56 percent of the respondents were directors and managers of 
research and evaluation. Skty-two percent of the respondents came from public school 
districts located in urban areas. One respondent came from a district in a lairal area, and 
one respondent represented an overseas school district for United States dependents 
(Figures 1 and 2). 

The districts in the sample varied in location and size. Ninety percent were categorized as 
one of the 500 largest public school districts in the United States (Sietsema, 1991). 
Twelve had more than 100,000 students and included the five largest districts: New York, 
Los Angeles, Chicago, Dade County, and Philadelphia. The districts represented each 
geographic region: Northeast, South, Midwest, and West. The South had the largest 
representation, 34 percent of the districts; the West, 29 percent; the Midwest, 24 percent; 
and the Northeast, 12 percent. Sixty-two percent of the districts were in urban areas 
(Figures 3, 4, and 5). 

!• What Is the Structure/Organization of Public School Districts' Testing Programs? 
How test data are reported. 

Test data were reported principally by grade level. The grade level information is 
organized and reported generally by district and by the school (98 percent of the districts). 
Next in frequency, test data were reported by individual students (88 percent) and then by 
classrooms (78 percent). 
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Figure 1. The percent of respondents and nonrespondents from 142 public school 
districts that were sent survey forms 
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Figure 2. Percent of public school district survey respondents by job title 
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Figure 3. The number and percent of public school districts surveyed by student 
enrollment 
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Figure 4. The percent of public school districts by census region: Northeast, South, 
Midwest, West, and other (Department of Defense Denpendents' Schools) 
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Over 90 percent of the districts tested their students at grades 3, 5, and 8. In contrast, very 
few, less than half, tested their students at prekindergarten, kindergarten, and grade 12 
levels. Six districts tested their students at all grade levels from prekindrrgarten to grade 
12 while most districts tested their students in nine to ten of the 14 grade levels. 

In order for school districts to follow individual student's academic progress from grade 
level to grade level, a continuous assessment program must be in place. From the survey 
information provided, 70 percent of the districts had grade level information for their 
students for at least nine of the 14 grade levels (Figures 6 and 7). 

Types of assessments. 

All but one (99 percent) of the public school districts tested their students with 
norm-referenced tests while 81 percent used criterion-referenced tests. Less than half (41 
percent) used district-wide performance-based assessment to assess academic achievement. 
To assess reading and mathematics, 98 percent of the districts used norm-referenced tests. 
Sixty-four percent assessed written language with a norm-referenced test and less than half 
(46 percent) of the districts assessed science with a norm-referenced test. 

The percentages for assessing these same subject areas dropped when the districts used 
criterion-referenced tests (CRTs). Less than three-fourths (71 percent) of the districts 
used CRTs for assessing reading, 78 percent for mathematics, 45 percent for written 
language, and a little over one-third (35 percent) for science. 

Performance-based assessment was used principally for written language assessment. 
However, this type of assessment involved only 36 percent of the districts. Overall, less 
than one-fifth of the districts used performance-based assessment for reading (17 percent) 
mathematics (13 percent), and science (9 percent). Regardless of the type of test, the 
assessment of science involved a very low percentage of the public school districts 
surveyed (Figures 8, 9, 10, and 11). 

2. What Capacity Do Public School Districts Have to Disaggregate Test Data by 
Demographics and Background Information and How Interested Are They in Doing It? 

By demographic and background information 

The process of disaggregating test data along different dimensions or variables involves in 
its simplest form cross tabulations. On the first level of analysis, cross tabulations by 
demographics allow program staffs, school administrators, and teachers to look at 
subgroups of students to determine their levels of academic achievement in the various 
subject areas, e.g., girls on the mathematics appHcations test at grade 3 or Hispanics boys 
on the reading vocabulary test in grade 6. 
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Figure 5. The percent of pubiic school district survey respondents hy urbanicity 
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Figure 6. The percent of public school districts that report test data by various 
organizational levels 
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Figure 7. The percent of public school districts that test at various grade levels, 
Prekindergarten t''^ Grade 12 
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Figure 8. The percent of public school districts that test students with norm- 
referenced tests, criterion-referenced tests, and performance-based 
assessments 
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Fieure 9 Percent of public school districts that use norm-referenced tests to assess 
^ students in reading, mathematics, science, language, and other subjects 



Norm-Referenced Data by Various Subjects 
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Figure 10. 



Percent of public school districts that use criterion-referenced tests to 
assess students in reading, mathematics, science, language, and other 
subjects 



Criterion-Referenced Data by Various Subjects 



Criterion 


100% 




80% 




o 

^ 60% 




C 

o 




5 40% 




20% 
0% 





78% 




Nr91 Public School Districts 



It is through this process of disaggregating data that the issues of disproportionalitj' arise 
and thus begins the analysis of students' oppcv^ unity to learn (Hathaway and Kershman, 
1988; Braddock, 1990; Stevens, 1990). We were surprised to learn that few districts 
disaggregated test data using demographic and background information although 82 
percent of the districts are capable of doing cross tabulations. The districts who did cross 
tabulation principally used only three variables: race/ethnicity, socioeconomic status, and 
gender. Sixty-six percent of the districts partitioned test data by student race/ethnicity, 37 
percent by socioeconomic status, and 14 percent by gender. The remaining student 
demographic variables were used by under 5 percent of the districts. 

District use of student and teacher background information to divide test data was 
extremely low. The percentages were under 5 percent with the exception of students' 
attendance rates with 13 percent of the districts using this student background variable. 
The void in implementing disaggregation of test data was not due to prohibitive school 
district policies. Only 4 percent of the public school districts had these policies (Figures 
12, 13, and 14). 

The positives and negatives of disaggregating test data 

In a follow-up interview with a subsample of the respondents, 18 persons were asked why 
their districts did or did not disaggregate test data. For those who did, their reasons 
centered on using the information to find the performance inequities of individual students 
and to focus on the needs of those students. The districts that did not disaggregate data 
(predominantly from districts in the South) indicated a fear that the data would be used 
destructively or in a negative political context. In particular, they felt that presenting test 
data along race/ethnicity lines might be seen as the principal outcome and not used to 
find out where resources should be placed to address student needs. 

The school districts that did disaggregate test data used the information internally and did 
not place it in reports to be disseminated to the public. However, 74 percent of the public 
school districts in the South reported that they disaggregated test data by race/ethnicity 
compared to 73 percent in the Midwest, 60 percent in Northeast, and 56 percent in the 
West. 

3, What Is the Interest of Public Districts to Use Oppjrtunity to Learn Variables and 
to Use Different Testing Processes in the Analysis of Student Data? 

Using opportunity to learn variables in cross tabulations 

Ninety-two percent of the districts did not use opportunity to learn variables when dividing 
test data into subgroups (Figure 14). This means that almost all of the districts did not 
investigate whether different groups of students did or did not have access to the content 
of the subject matter and the content of the test. Their information was limited to 
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Figure 11. The percent of public school districts that use performance-based 

assessments to assess students in reading, mathematics, science, language, 
and other subjects 
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Figure 12. The percent of public school districts that analyze test data by 
demographics 
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Figure 13. The percent of public school districts that report test data by teachers' and 
students' background information 
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Figure 14. The percent of public school districts thu^ have the capacity to do cross- 
tabulations, have policies that prohibit cross-tabulation analysis, and use 
opportunity to learn data in cross tabulation analysis 
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student demographic and/or background information. There was no information to see if 
achievement was in any way attributed to teachers' instructional practices. 

In follow-up interviews, all of the respondents indicated that variables on opportunity to 
learn should include the analysis of student outcome data such as test scores. There was a 
general feeling among those interviewed that using variables on opportunity to learn with 
student outcome data would help explain student performance and improve instruction. 

Validating test score accuracy 

Teacher accountability in relation to student performance was seen as a serious issue. For 
example, it was suggested that student achievement should be reviewed at the teacher 
level. One respondent stated, 'The whole issue of opportunity to learn brings forward the 
question to what extent has instruction occurred." Respondents stressed the unfairness in 
comparing classes, schools, or school districts by test scores without ensuring that the 
content was covered equitably. Without information on opportunity to learn, test scores 
are meaningless and subject to misinterpretation. However, they cautioned that 
information on opportunity to learn should not be used to make excuses but used instead 
to overcome learning deficiencies. One respondent indicated that it was necessary to 
analyze test data with data on opportunity to learn to move the emphasis from the 
"victims" to those delivering instruction. 

Measuring opportunity to learn 

In follow-up interviews, respondents expressed one major reservation about using variables 
on opportunity to learn to analyze test data. They questioned whether these variables 
could be quantified in order to conduct the analysis. These expressed concerns pointed 
out that some of the respondents lacked information about analyzing content coverage, 
curriculum content and test content overlap, and content emphasis found in previous 
research studies on opportunity to learn. The previously cited studies on content 
coverage, content exposure, and content emphasis showed that opportunity to learn 
variables are measurable. However, Wyatt (1991) cautioned that the current measures of 
opportunity to learn need to be improved and new measures need to be developed. 

The advantage of interval/segmented testing 

Interval or segmented testing divides curriculum assessment into smaller sections or 
segments versus an end-of-year test that samples the whole year's curriculum. According 
to the survey, only 23 percent of the public school districts used this assessment process 
while 54 percent indicated that they did not. However, a more telling statistic was that 
only 6 percent or 5 of the 91 districts have future plans to implement interval or 
segmented testing. For this particular question, 64 percent of the respondents gave no 
responses. This high rate of nonresponse may indicate that interval or segmented testing 
was a nev/ topic and not previously considered or discussed in their districts (Figure 15). 
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The values of interval/segmented testing was demonstrated in the Pittsburgh and Los 
Angeles public school districts (LeMahieu, 1983; Wasney, 1990, 1989). The districts used 
this type of assessment to monitor students' progressive mastery of subject matter skills. 

In Pittsburgh, the Monitoring Achievement in Pittsburgh (MAP) project involved the 
periodic assessment of students in mathematics every 6 weeks. Twenty broad terminal 
objective/skills and their corresponding 20 items were used in the MAP. There was one 
item per skill. If a student responded correctly to an item, it was assumed that the skill 
was mastered. This monitoring program seemed a powerful tool in enhancing the 
achievement of the students. At all grades, the increase in student performance was much 
greater in the overlapped areas (curriculum content overlapped the MAP items) than in 
the non-overlapped regions of the curriculum. Also, it proved to be a method to monitor 
student opportunity to learn the subject matter through access to the core curriculum. 

The Los Angeles Mile Post Testing Program was modeled after MAP. The program 
differed in that the subjects monitored were limited to reading, mathematics, and 
language, and the assessment period was every 8 weeks. Also, the students were tested on 
small groups of items per skill to determine their mastery of the skills, and hopefully, to 
allow for reteaching by modifying instructional strategies and/or providing remediation 
activities (e.g., tutoring) for the students. 

4. For Public School Districts, Is the Opportunity to Learn Definition Sufficient? Should 
the Definition Be Expanded? 

In response to these survey questions, 41 percent or 37 of the 91 respondents offered 
suggestions on how to expand the definition while 59 percent did not. Because several 
respondents appeared uncertain about what opportunity to learn was, they found it 
difficult to suggest expansions of its definition. One person said, "I have been trying for 
some time to integrate several concepts in my head and opportunity to learn seems to 
help this. Specifically, I have been concerned about retention-in-grade, mastery learning, 
and teaching on the students' instructional level. I believe these are three facets of the 
same problem: We withhold new information from students who haven't mastered some 
set of objectives." Another respondent commented that, "I haven't read much about 
opportunity to learn until your survey came. It has raised my awareness level of (what is) 
opportunity to learn." 
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Figure 15. The percent of public school districts that currently have 

interval/segmented testing and the '^^.rcent that have future plans for 
implementing interval/segmented tooting 
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Including quality of instructional delivery. 

Thirty-seven respondents made 62 suggestions to expand the definition of opportunity to 
learn- Their suggestions could be sunmiarized into six categories: content coverage; 
content exposure; content emphasis; quality of instructional delivery; students* readiness to 
leam; and school environment and climate conducive to learning. Four of the six 
categories are teacher-focused and relate to the issues of this paper. The largest number 
of the suggestions was for the category entitled quality of instructional delivery. This 
category received one-third of the suggestions. 

The quality of instructional delivery category differed from the traditional definition of 
opportunity to leam that encompasses the elements of: content coverage, content 
exposure, and content emphasis. Quality of instructional delivery contains these 
suggestions for teachers: teach coherent lessons; teach grade level curriculimi skills; use 
varied and appropriate teaching strategies; teach higher levels of skills; and know the 
subject matter content (Table 3). District level researchers indicated that this category was 
vital to the investigation of students* opportunity to leam. 

Follow-up interviews validated further the survey responses about adding the quality of 
instructional delivery to the definition of opportunity to leam. The respondents strongly 
commented on the need for teachers to know the content that they were teaching. Also, 
they were concerned about teachers being able to handle the curriculum. "Some teachers 
are having on-the-job-training (to teach)." They stressed the need to upgrade the skills of 
some teachers so they can "do what is appropriate" and "have more sldlls and approaches 
to teaching." 

We acknowledge that this particular category would be hard to measure. However, those 
responsible for reporting student outcome data stressed the importance of having and 
using this information. This finding certainly indicates that the investigation of quality of 
instructional delivery cannot be minimized because of the scope of the curriculum and the 
difficulty of the task. 
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Table 3. Respondents suggestions to expand the definition of opportunity to learn 



Category/Element 



Content Coverage 

o evidence of content and test overlap 

o access to the core curriculum 
Content Exposure 

o time-on-task 

o quantity of instruction provided in the subject area 
Content Emphasis 

o expectations of students' capacity to learn and achieve 

o provision of emotional support for students to learn 
Quality of Instructional Delivery 

o teaches grade level curriculu: 

o uses appropriate teaching techniques and strategies (e.g., has variety of approaches, mtroduces new 
skills, reviews skills taught, gives appropriate feedback to students, etc) 
o knowledgable on subject matter content (e.g., subject credentials, professional experiences) 
o teaches subject matter content to students' mastery 
o commits to teaching (e.g., attendance rate) 
Students' Readiness 

o has English language proficiency 

o has language proficiency relative to the subject matter 

o attended preschool and/or kindergarten 

o has prior instruction and knowledge in the subject area 

o attitudes towards learning, school and self 

o commitments to schooling (e.g., attendance rate and transiency rate) 
School Environment and Climate 

0 access to challenging courses and elective choices 

o provides quality (appropriate) instructional material 

o provides sufficient quantity and variety of instructional materials 

0 provides special programs for students 

o no pull-out instructional programs for some students 

o class size 

Note: Thirty-seven respondents made 62 suggestions to expand the opportunity to learn definition. 



30 



kJ I. 



What Do These Findings Tell Us 



Opportunity To Learn is Virtually An Unknown Concept in the United Stages. 

As opportunity to learn continues to grow in importance in international studies of 
academic achievement among countries, the recognition of the importance of using 
opportunity to learn information in public school districts in the United States has not 
fully emerged. In fact, at the time of the survey, it was almoFt an unknown concept to 
most of those responsible for analyzing and reporting student outcome data in public 
school districts. 



Students* Differences in Academic Achievement Are Not Being Related to an Analysis of 
Opportunity to Learn. 

Opportunity to learn information was not used to question the differences in academic 
achievement and was not used as a variable to analyze the differences in student outcome 
data. This was not because public school districts did not have the technical capabilities to 
do so. In fact, most districts were fully capable, but few chose to implement the process. 
Instead, they reported test data principally by grade level and organizational level, e.g., 
district, region, school. 

Most Districts Limit the Disaggregation of Test Data to Race/Ethnicity. 

The only variable used by over two-thirds of the districts to report test data was 
race/ethnicity. Although divided by subgroups, the data provided no insight into why 
these racial groups were achieving differently. This was sufficient if the districts 
subscribed to the notion that race was the difference. However, several of the respondents 
in the districts did not disaggregate data by race/ethnicity because of the fear that this 
type of analysis would be the major outcome. 

Lack of Opportunity to Learn Information Hampers Teachers* Abilities to Improve Their 
Teaching Practices. 

Realistically, without analyzing opportunity to learn information, districts using only 
demographics to disaggregate test data have insufficient information for instructional 
decisionmaking. Teachers do not group students by race/ethnicity. Instead, they want to 
focus their teaching practices based on students* learning needs. Presently, little is known 
about what is actually happening in classrooms that causes some students to achieve and 
others not to achieve. 
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As pointed out earlier, previous research studies and survey and interview respondents 
confirm that investigating whether or not the teachers provided equal access to the 
curriculum to all students, provided exposure to the subject matter, and emphasized what 
was important in the curriculum, shifts the responsibility for learning from the students to 
the teachers. This finding was underscored in several small comparative studies of public 
schools in Japan, Taiwan, and the United States. 

Stevenson and Stigler and their associates moved beyond race (they found no significant 
differences in cognitive abilities attributed to race) as the explanation for the students in 
Japan and Taiwan performing better in mathematics than U.S. students (1982, 1985, 1986, 
1990, 1991, 1992). They found several important factors related to opportunity to leara 
that contributed to the differences in student achievement. 

One factor was the greater amount of time devoted to mathematics instruction in both 
Japan and Taiwan, compared to the U.S. classrooms. Another was that children in Japan 
and Taiwan spend more time on homework than children in the U.S. Related to this, 
American parents do not help their children as much with their homework. Next, it was 
noted that learning requires time and practice. When either is reduced, learning is 
impaired. American children spend less time in academic activities, than the children in 
the other countries, and Japanese children were more likely to be attentive to their 
teachers. 

Regarding curriculum and instruction, curricula are precisely defined and followed in 
Japan and Taiwan; whereas, U.S. teachers are allowed to decide what to emphasize 
according to their desires. In Japan and Taiwan, teachers lead their students more than 
two-thirds of the time while U.S. children are led less than half the time. U.S. teachers 
spend more time giving directions than imparting information. Based on these 
observations, it would appear that U.S. children do not receive the same amount of 
instructional time in mathematics as do the children in Taiwan and Japan. 

External to what teachers do in the classroom but related to their instructional impact was 
information about parental beliefs. According to the studies, parents who emphasize 
ability as the most important requisite for academic success may be less disposed to stress 
the importance of effort. In Japan, more importance is placed on effort and working 
hard; whereas, in the United States, parents assign more importance on ability. 



Using Data on Opportunity to Learn to Improve Teaching* 

From the information gleaned ftom the international and national studies, collecting 
information on content coverage and content emphasis is an integral equity practice and 
should occur as part of the teaching and learning paradigm. All of the public school 
persons responsible for reporting test data indicated that using opportunity to leara 
variables to analyze student outcome data would be a tremendous breakthrough in 
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assisting school administrators and teachers to correct and improve teaching and to focus 
the placement of humari and material resources where they are needed the most. 

Most School Districts Lag Behind on Analyzing Information on Opportunity to Learn. 

Right now, some public school districts attempt to improve the academic achievement of 
their students by using test data disaggregated by only a few demographic variables. 
However, the majority of the public school districts do not use any disaggregated 
information. The question about disaggregating outcome data was necessary because this 
process could be the first step in implementing an analysis of opportunity to learn 
information. 



Opportunity to Learn Science Cannot Be Analyzed Because Science Is Not Tested. 

It is generally believed that district-wide assessment of certain subject matter influences 
teachers in what they teach. On a national basis, science is seen as one of the major 
subjects for the future competitiveness of students leaving public schools and entering the 
workforce. This may be true, but the importance of science was not reflected in reports of 
what was being tested in the districts we surveyed. If the old statement, "We teach what is 
tested" is true, science instruction in public schools faces a gloomy future. 

Performance-based Assessment Is a Limited Process in Schools. 

Performance based assessment is another process that has not fully arrived in the public 
schools as far 3S district-wide testing is concerned. Surprisingly, this included written 
language assessment where only a little over one-third of the districts used 
performance-based assessment to determine academic achievement. 

The Traditional Assessment Mode Continues to Be End-Of-Year Norm-Referenced 
Testing. 

Presently, studying the academic progress of students involves two strands. First is the 
year-to-year progress that necessitates student assessment at each grade level. We learned 
that the preferred mode for this assessment is norm-referenced testing followed by 
criterion-referenced testing. 

Second is the progress monitored during the school year This is usually limited to 
teacher quizzes and tests given within their classrooms. There are no comparisons to 
student progress in other classrooms* During the school year, more meaningful progress 
information could be garnered by using grade level or subject matter interval/segmented 
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testing. Teachers then know what is happening external to their classrooms throughout 
the year to segments or groups of grade level skills and/or subject matter concepts. 
However, most districts relied only on end-of-year assessment information to make 
decisions about student progress. Less than one-fourth of the public school districts used 
interval/segmented testing; and only 6 percent indicated that this process was in their 
future plans. This amounted to 5 of the 91 pubHc school districts. 

InteiTval/Segmented Testing Can Be Used For Determinik^g Students^ Opportunity to 
Learn. 

In an evaluation of ten schools in Los Angeles, it was determined that skills and objectives 
that were taught and then mastered by large percentages of students on the 
interval/segmented tests later showed similar rates of mastery on the end-of-year norm- 
referenced test. This same finding was reported for MAP in Pittsburgh. This monitoring 
process is closely related to opportunity to ieam in that it provides information about 
content coverage and content emphasis. 

The tediousness of some of the measurement procedures suggested in previous 
opportunity to learn research reports could be simplified wiih shorter numbers of items to 
analyze over shorter periods of time. In addition, the clustering of skills into smaller 
segments of time for assessment (e.g., 6 or 8 weeks) allows for quicker attention to the 
remediation of the skills not mastered. 



Quality of instructional Delivery Is a Valuable Addition to the opportunity to Learn 
Definition. 

Last, but not least, were the suggestions to expand the definition of opportunity to leara to 
include the more difficult concept or variable of quality of instructional delivery. Several 
researchers who are interested in opportunity to learn as a research topic (Leigh Burstein 
of UCLA, Lorraine McDonnell of Rand Corp and U.C. Santa Barbara, and Andy Porter 
of the University of Wisconsin) have talked about the difficulty of measuring this variable. 
However, the respondents in the public school districts felt that in addition to content 
coverage, this was a critical variable to use when analyzing students* opportunity to leam 
in their schools. 

Although not expressly called "quality instructional delivery," there exists such research by 
Stevenson and Stigler and their associates on the international level and from Brophy and 
Good and the researchers at the Center for Research on Effective Schooling for 
Disadvantaged Students at Johns Hopkins University on the national level. The Center's 
recent research emphasizes the importance of looking at quality instructional deliveiy for 
disadvantaged students. Also to be noted is the emerging research on teaching excellence 
from the National Board for Professional Teaching Standards. 
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On the issue of quality instructional delivery, Jones (1991) points out that there are new 
visions of learning, curriculum, instruction, and assessment and that schools-teachers and 
administrators-should be held accountable for using the '^best practices*** Teachers should 
be provided with a repertoire of instructional strategies that are effective for teaching all 
students. 

As Wyatt (1991) indicated in his monograph, investigating the various methods of 
determining opportunity to leara is needed to get the best picture of what is happening in 
the different classrooms. This investigation should include the quality of instructional 
delivery as an important element in improving the academic achievement of poor and 
minority students in the United States, 
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What Are the Policy Recommendations? 



To be effective for all students, teachers need to teach differently (Kennedy, 1991). The 
public school districts, particularly inner city urban school districts, now serve increasing 
numbers of poor and minority students. Teachers need to learn how to effectively teach 
the diverse s: adent populations (poor and minority) now present in the public schools. 
Business as usual is unproductive and destroys the potential of these students. New 
instructional practices that are effective for poor and minority students should also be 
beneficial for all students. However, we will not know the true effects of the old and new 
practices for different groups of students until opportunity to learn is investigated when 
analyzing student outcome data. Because of this, there are certain policy 
recommendations related to the findings of this study that need to be addressed. 

Two time frames exist for this set of recommendations. Some can be addressed 
immediately while others will require structural revisions of procedures and organization 
within institutions. 

First: Raise the Awareness and Knowledge Levels of All Parties Who Are Responsible for 
Educating Students. 

1. Just as there was a summit to develop national goals, a summit should be called 
by the Department of Education to emphasize the need to teach all curriculum to all 
students and to improve teaching practices in our public schools. The invitees should be 
the leadership of the American Federation of Teachers (AFT), the National Education 
Association (NEA), The National Board for Professional Teaching Standards, 7h Council 
of Great City Schools, the organization for schools of education in our colleges and 
universities, and others. They should convene to learn about and discuss the issues of 
opportunity to learn with those international and national researchers who have worked in 
this area. 

2. Public school districts must become more aware of the need to investigate 
' students' opportunities to learn when reporting student outcomes and recommending 

courses of action to address these outcomes. Reporting outcome data by race, 
socioeconomic status, and gender does not provide the information needed to 
substantively improve student outcomes. This tells policymakers where the problems are 
but it does not tell administrators and teachers what to do better in the area of instruction 
and classroom practices. To focus on where to initiate appropriate change in the schools, 
classrooms, and schools of education, policymakers should require the reporting of 
information on opportunity to learn with other student outcome data. 

3. The U.S. Department of Education and supportive educational organizations 
should organize meetings across the nation to promote the concept of opportunity to learn 
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to administrators and teachers in an effort to bring about change beneficial to all students 
in our classrooms. 



Second: Provide Training for Public School District Personnel to Use Opportunity to 
Learn Information, to Encourage the Collection of Opportunity to Learn Data^ and to Do 
More Research on the Quality of Instructional Deliveiy. 

4. District directors of research, evaluation, and assessment, information technology 
specialists, curriculum specialists, teachers, and administrators should be trained to collect 
and analyze opportunity to learn information. Federal and state agencies should assist 
public school districts in training their staffs. Without technical assistance, it is unlikely 
that opportunity to learn data will be collected and analyzed. On a similar note, 
preservice training of new teachers and administrators in schools of education in colleges 
and universities should include opportunity to leara as part of their postsecondary 
education reform efforts. 

5. The importance of opportunity to learn information will be recognized by public 
school districts when federal and state data collection activities include requests for 
information on opportunity to learn. All new and continuing elementary and secondary 
studies should include opportunity to learn information. 

6. Researchers need to develop "user friendly" methods for determining student 
opportunity to learn. NCES should fund this type of research to encourage public school 
districts to collect and use the information to facilitate the Center*s collection of this 
information for federal reports. 

7. Research should be conducted on how to operationally define quality 
instructional delivery better so that survey and other data collection activities can be 
developed to cover this element of opportunity to learn. Again, NCES should fund this 
research to facilitate the collection of data for their elementary and secondary reports that 
provide information to other federal agencies. 



ERIC 



38 



Conclusion 



Consequences of not changing the way we look at student outcome data have been 
enumerated many times over, e.g., tracking, dropping out of school, and so on. Research 
results that reported higher academic achievement by students of other nations generated 
many questions. Policymakers and researchers were encouraged to expand their collection 
of opportunity to learn information in an attempt to understand the reasons for the 
achievement differences. Similarly, policymakers and educators cannot be complacent 
with the reports of student outcomes based solely on race, socioeconomic status (SES) and 
gender. Just as the international studies looked beyond race, SES, and gender and looked 
at the instructional and cultural practices in different countries, researchers must do the 
same when looking at different groups of students in the United States. Investigating 
students' opportunity to learn is a viable way to determine if equity exists for all students 
in our public schools. 



39 



References 



Adams, M. (1990). Beginning to read: Thinking and learning about print. Cambridge, 
MA.: MIT Press. 

Bloom, B. (1976). Human characteristics and school learning. New York: McGraw-Hill. 

Bobbitt, S. (1992). Filling the gaps: An overview of data on education m grades K 
through 8. National Center for Education Statistics. Washington D.C. Forthcoming. 

Bracey, G. (1991). Why can't they be like we were? Phi Delta Kappan . 73, (2), 104-117. 

Braddock, J. (1990). Tracking implications for student race-ethnic subgroups. Center for 
R -^search on Effective Schooling for Disadvantaged Students, No. 6. Baltimore: The 
Johns Hopkins University. 

Brophy, J. and Good, T. (1986). Teacher behavior and student achievement. In M. 
Wittrock (Ed.). Handbook of research on teaching. New York: MacMillan Publishing 
Company. 

Burstein, L., Guiton, G., Bayley, L., and Isaacson, A. (1991). Survey of mathematics and 
science opportunities: Compilation of items measuring mathematics and science 
opportunities and classroom processes from large-scale educational surveys. CRESST, 
Research Report Series No. 1. Los Angeles: Graduate School of Education, University of 
California. 

CPRE Policy Briefs (1991). Equality in education: Progress, problems, and possibilities. 
Eagleton Institute of Politics. New Brunswick, NJ.: Rutgers, The State University of New 
Jersey. 

Coleman, J., Campbell, E., Hobson, C, McPartland, J., Mood, A., Weinfield, F., and York, 
R. (1966). Equality of Educational Opportunity. Washington: U.S. Government Printing 
Office. 

Doyle, W. (1983). Academic Work. Review of Educational Research . 53 (2), 159-199. 

Education Counts: An indicator system to monitor the nation's educational health. (1991). 
Washington, D.C: National Center for Education Statistics. 

Floden, R., Porter, A., Schmidt, W., Freeman, D., and Schwille, J., (1981). Responses to 
curriculum pressures: A policy-capturing study of teacher decisions about context. 
Journal of Educational Psychol ogy. 73, 129-141. 



41 

d.: 



Goldenberg, C. and Gallimore, R. (1991). Local knowledge, research knowledge, and 
educational change: A case study of early Spanish reading improvement. Educational 
Researcher, 20 (8), 2-14. 

Goodlad, J. (1983). Access to knowledge. Teachers College Record, 84 (4), 787-819. 

Goodlad, J* and Oakes, J. (1988). We must offer equal access to knowledge. Educational 
Leadership, 45 (5), 16-22. 

Hathaway, W, and Kershman, J. (1988). Down with disproportionality: Rising to the 
challenge of differential school performance among student ethnic groups. Portland, OR: 
Portland State University/Portland Public Schools Center for Urban Research. 

International Association for the Evaluation of Educational Achievement (lEA) (1982). 
Second study of mathematics: Teacher questionnaires. 

Jones, B. (1991). Students at risk vs. the board of education: Toward formulating better 
practices and policies. Unpublished draft paper from the North Central Regional 
Laboratory. 

Kennedy, M. (1991). Policy issues in teacher education. Phi Delta Kapp an, 72 (9), 
659-665. 

Kirst, M. (1991). The need to broaden our perspective concerning America's educational 
attainment. Phi Delta Kappaa 73 (2), 118-120. 

Leinhardt, G. (1983). Overlap: Testing whether it is taught. In G.F. Madaus (Ed.) The 
courts, validity, and minimum competency testing. Boston: Kluweer-Nijhoff. 

Leinhardt, G., and Seewald, A. (1981). Overlap: What's tested, what's taught? Journal 
of Educational Measurement, 18 (2), 85-96. 

LeMahieu, P. (1983). The effects on achievement and instructional content of a program 
of student monitoring through frequent testing. Educational Evaluation and Policy 
Analysis, 6 (2), 175-187. 

LeMahieu, P., and Leinhardt, G. (1985). Overlap: Influencing what's taught, a process 
model of teachers' content selection. Journal of Classroom Interaction, 21, (1), 2-11. 

McDonnell, L., Burstein, L., Catterall, J., Ormseth, T., and Moody, D. (1990). Discovering 
what schools really teach: Designing improved course-work indicators. Santa Monica, 
CA: Rand Corp. 

Murphy, J. (1988). Equity as student opportunity to learn. Theory into Practice, XXVII, 



42 

d 



(2), 145-151. 



Oakes, J. (1983). Tracking and ability in American schools: Some constitutional 
questions? Teachers College Record , 84 (4), 801-819. 

Oakes, J. (1990). Multiplying inequalities: The effects of race, social class, and tracking 
on opportunities to learn mathematics and science. Santa Monica, CA: Rand Corp. 

Outcomes of Education. (1991). OECD International Education Indicators Network A: 
Report of Phase 2. Paris, France: Organization for Economic Cooperation and 
Development. 

Orland, M. (1990). Demographics of disadvantage: Intensity of childhood poverty and its 
relationship to educational achievement. In J. Goodlad and P. Keating (Eds.). Access to 
knowledge: An agenda for our nation's schools. New York: College Entrance 
Examination Board. 

Schmidt, W. (1990). The distribution of instructional time to mathematical content: One 
aspect of opportunity to learn. Michigan State University, (in press). 

Shavelson, R. (1976). Teachers' decision making. In N.L. Gage (Ed.). The psychology of 
teaching methods: The 75th Yearbook of the National Society for the Study of Education, 
Part 1. Chicago: University of Chicago Press. 

Shavelson, R. and Stern, P. (1981). Research on teachers' pedagogical thoughts, 
judgments, decisions, and behaviors. Review of Educational Research, 5 (4). 

Sietsema, J. (1991). Characteristics of the 100 Largest Public Elementary and Secondary 
School Districts in the United States: 1988-1989. Washington, D.C.: National Center for 
Education Statistics. 

Sirotnik, K. (1990). Equal access to quality in public schooling: Issues in the assessment of 
equity and excellence. In J. Goodlad and P. Keating (Eds.). Access to knowledge: An 
agenda for our nation's schools. New York: College Entrance Examination Board. 

Stevens, F. (1990). Testing: Current practices and policy issues in the education of at-risk 
students. A paper presented at the One-Third of a Nation Conference. Washington, 
D.C., Howard University. 

Stevenson, H., Lee, S., and Stigler, J. (1990). Making the grade in mathematics. Reston, 
VA: National Council of Teachers of Mathematics. 

Stevenson,.H., Stigler, J., Lee, S. Lucker, G., Kitamura, S., and Hsu, C. (1985). Cognitive 
performance and academic achievement of Japanese, Chinese, and American children. 

43 



Child Development . 56, 718-734. 



Stevenson, H., Stigler, J., and Lee, S. (1986). Achievement in mathematics. In H. 
Stevenson, H. Azuma, and K. Hakuta (Eds.). Child Development in Japan. New York: V/. 
H. Freeman and Company. 

Stevenson, H. and Stigler, J. (1992). The learning gap: Why our schools are failing and 
what we can learn from Japanese and Chinese education. New York: Summit Books. 

Stigler, J., Lee, S., Lucker, W., and Stevenson, H. (1982). Curriculum and achievement in 
mathematics: A study of elementary school children in Japan, Taiwan, and the United 
States. Journal of Educational Psycholog^ 74, 315-322. 

Stigler, J. and Stevenson, H. (1991). How Asian teachers polish each lesson to perfection. 
American Educator , 12-47. 

Walker, D. and Schaffarzick, J. (1974). Comparing curricula. Review of Educational 
Research, 44, 83-111. 

Wasney, T, (1989). Ten schools evaluation reports: Interim achievement report. 
Publication No. 555, Part 111, A and B. Los Angeles: Los Angeles Unified School District. 

Wasney, T. (1990). Ten schools evaluation reports: Interim academic achievement report. 
Publication No. 569, Part 1 B. Los Angeles: Los Angeles Unified School District. 

Wiley, D. (1990). Opportunity to learn: A briefing for the Advisory Council on Education 
Statistics, National Center for Education Statistics. Washington D.C. 

Winfield, L. (1987). Teachers, estimates of test contest covering in class and first-grade 
students, reading achievement. The Elementary School Journal , 87 (4), 438-45. 

Wyatt, T. (1991). The concept and measurement of opportunity to learn in the 
international context Draft for publication in the University of Sydney Monograph series. 

Yoon, B., Burstein, L., Gold, K., Chen, Z., and Kim, K. (1990). Validating teachers, 
reports of content coverage: An example from secondary school mathematics. A paper 
presented at the Annual Meeting of the National Council on Measurement in Education. 
Boston, Massachusetts. 



44 

ERIC 



APPENDIX A 



Table A. The number and percentag of public school districts in the study by location, 
enrollment size, and region. 



District Characteristic Number Percent 



All 

All 


Q1 

y i. 


100 

1 \J\J 


Location of the district 






Urban 


56 


62 


Suburban 


33 


36 


Rural 


1 


1 


Other 


1 


1 


Enrollment size 






< 12,125* 


9 


10 


12,125-- 19,999 


11 


12 


20,000- 29,999 


10 


11 


30,000- 39,000 


11 


12 


40,000- 49,999 


12 


13 


50,000- 59,000 


7 


8 


60,000- 69,000 


12 


13 


70,000- 79,000 


3 


3 


80,000- 89,000 


1 


1 


90,000- 99,000 


3 


3 


< 100,000 


12 


13 



*Note. Student enrollment above 12,125 comprised th3 500 largest public school districts 
in the United States, 1988-89. There were 177 public school districts in the United States 
with student enrollment above 25,000 students. 

Percentages may not add to 100 due to rounding. 



Table B. Public school districts by census regions: northeast, south, midwest, and west. 



Region 


Number 


Percent 


Northeast 


11 


12 


South 


31 


34 


Midwest 


22 


24 


West 


26 


29 


Other 


1 


1 


Total 


91 


100 



Table C. Number and percent of respondents by public school district job title. 



Title Number Percent 

Assistant superintendent 8 9 

Director/manager 51 56 

Assistant director 3 3 

Test Coordinator/supervisor 18 20 

Program evaluator/researcher 11 12 

Total 91 100 



46 

4S 



Table D. Number and percent of public school districts that test grades pre-k through 12. 



Yes No 



Grade Level 


Number 


Percent 


Number 


Percent 


Pre-kindergarten 


/ 


o 
O 


QA 


00 
yz 


Kindergarten 


2o 


zy 




/ 1 


1 


j5 


OU 


JO 




2 


'71 
/I 


/o 




zz 


3 


82 


90 


9 


;o 


4 


77 


85 


14 


15 


5 


84 


92 


7 


8 


6 


76 


84 


15 


16 


7 


78 


86 


13 


14 


8 


80 


88 


11 


12 


9 


62 


68 


29 


32 


10 


63 


69 


28 


31 


11 


55 


60 


36 


40 


12 


32 


35 


59 


65 



Note: N=91. 
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Table E. Number and percent of public school districts that collect norm-referenced, 
criterion-referenced, and performance-based data. 



Yes No 
Type of test data Number Percent Number Percent 

Norm-referenced 90 99 

Criterion-referenced 74 81 16 18 

Performance-based 41 45 46 51 



Note: No response: NRT=1; CRT=1; and PBA=4. 



Table R Number and percent of public school districts tliat collect norm-referenced 
data by subject area. 



Subject 

Reading 

Mathematics 

Science 

Language (oral) 
Language (written) 
Other* 



Yes 



Number 


Percent 


89 


98 


90 


99 


46 


51 


4 


4 


64 


70 


25 


28 



No 



Number 


Percent 


2 


2 


1 


1 


45 


49 


87 


96 


27 


30 


66 


72 



*Note. Other includes the following subjects: social studies (21), study skills (2), 
spelling (3). 
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Table G. Number and percent of public school districts that collect criterion-referenced 
data by subject area. 



Subject Number 

Reading 65 

Mathematics 71 

Science 32 

Language (oral) 6 

Language (written) 41 

Other* 18 



No 



Percent 


Number 


Percent 


71 


26 


29 


78 


20 


22 


35 


59 


65 


7 


85 


93 


45 


50 


55 


20 


73 


80 



*Note: Other includes the following subjects: social studies (13), end-of-course (1), 
foreign language (1), health (4), citizenship (2), study skills (1), home economics (1). 



Table H. Number and percent of public school districts that collect performance-based 
data by subject area. 



Type of subject Number 

Reading 15 

Mathematics 12 

Science 8 

Language (oral) 5 

Language (written) 33 

Other* 6 



No 



Percent 


Number 


Percent 


17 


76 


83 


13 


79 


87 


9 


83 


91 


6 


86 


94 


36 


58 


64 


7 


85 


93 



*Note. Other includes the following subjects: social studies (2), composition (2). 
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Table L Number and percent of public school districts that provide test data by 
organizational levels. 



Organizational 




Norm 


level 


N 


% 


District 


89 


98 


Region/ Area/Zone 


24 


26 


School 


89 


98 


Classroom 


71 


78 


Students (special 


53 


58 


groups) 






Students 


80 


88 


(individuals) 







test data 



Criterion 


Performance 


N % 


N 


% 


68 75 


28 


31 


19 21 


3 


3 


69 76 


28 


31 


69 66 


17 


19 


34 37 


13 


14 


65 71 


33 


36 



Note. N = 91. 



Table J. Number and percent of public school districts that report test data by 
demogtraphic characteristics of students. 



Demographic Number Percent 

Race/ethnicity 60 66 

SES (or surrogate SES) 34 37 

Parents' educational level 4 4 

Parents' job status 1 1 

Other* 28 31 



*Note. Other includes the following: gender (22), language proficiency (2), post high 
school plans (1), special programs (6), repeater status (1), handicapped (1), Chapter 1 
(3), time in the district (2), cohort membership (1). 
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Table K. Number and percent of public school districts that report test data by teacher 
and student background information. 



Background 


Number 


Percent 


Years of teaching 


2 


2 


experience 






Teachers' attendance 


4 


4 


rate 






Students' attendace 


12 


13 


rate 






Other* 


12 


13 



•Note. Other includes the following: type of magnet (1), limited edition speaking 
students (1), special education students (2), anticipated grade in the course (1), grade 
level (1), years in the school (1), years in the district (2), years in the country (1), school 
SES (1), special programs (1), student mobility (3), parental participation (1), 
achievement gain from prior year (1), English language proficiency (1). 
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Table L, Number and perdent of school districts which have district-managed interval or 
segmented testing program, have future plans to have an interval or segmented testing 
program. 



Yes No No 

Response 

Item N % N % N % 

Has interval/segmented 21 23 49 54 

testing 

o Norm-referenced 

format 8 9 83 91 - - 

o Criterion- 
referenced format 15 16 76 84 

Future plans 

for interval/ 5 6 28 31 58 64 

segmented testing 



Note. N = 91, 



52 



Table M. Number and percent of districts that have the capacity to do cross tabulations, 
have district policies which prohibit cross tabulations by test data and demographic and 
background data, and use OTL as a crosstab in their data analysis. 



Yes No 

Item Number Percent Number Percent 

District office has the 82 16 18 

capacity to do cross 
tabulations with 
demographic and 
background data. 

District policy would ^ 4 85 93 

prohibit cross tabulations of 
test data with demo- graphic 
and back-ground data. 

OTL is used as a cross-te ^ 3 ^ 

in data analysis. 



Note. N = 91. No response for policy=2, 0TL=2. 



Table N. Number and percent of respondents and nonrespondents from sample public 
schools. 



Districts Number Percent 

All 142 100 

Respondents 91 64 

Nonrespondents 51 36 
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Table O. Participating Public School Districts by Region 



Region/District 



City/State 



Northeast (11) 
Boston Public Schools 
Buffalo Public Schools 
East Windsor Regional School 
Hartford School District 
New York City Public School 
Newark City School District 
School District of Philadelphia 
Pittsburgh Public Schools 
Rochester City Schools 
Springfield Public Schools 
Troy School District 

Midwest (22) 

Ann Arbor Public Schools 

Black River Falls 

Chicago Public Schools 

Cincinnati Public School 

Cleveland Schools 

Columbus Public Schools 

Dayton City Schools . 

Dearborn Public Schools 

Des Moines Public Schools 

Detroit City School District 

Evanston Public Schools 

Grand Rapids Public Schools 

Independent School District #196 

Kalamazoo Public Schools 

Kansas City, Missouri Public Schools 

Madison Metropolitan School District 

Milwaukee Public Schools 

Omaha Public Schools 

Pontiac School District 

Saint Paul Public Schools 

Toledo Public Schools 

Wichita Public Schools 



Boston, Massachusetts 
Buffalo, New York 
Hightstown, New Jersey 
Hartford, Connecticut 
New York, New York 
Newark, New Jersey 
Philadelphia, PA 
Pittsburgh, PA 
Rochester, New York 
Springfield, MA 
Troy, New York 



Ann Arbor, Michigan 
Black River Falls, m 
Chicago, Illinois 
Cincinnati, Ohio 
Cleveland, Ohio 
Columbus, Ohio 
Dayton, Ohio 
Dearborn, Michigan 
Des Moines, Iowa 
Detioit, Michigan 
Evanston, Illinois 
Grand Rapids, Michigan 
Rosemount, Minnesota 
Kalamazoo, Michigan 
Kansas City, Missouri 
Madison, Wisconsin 
Milwaukee, Wisconsin 
Omaha, Nebraska 
Pontiac, Michigan 
Saint Paul, Minnesota 
Toledo, Ohio 
Wichita, Kansas 
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Table O. Participating Public School Districts by Region (continued) 



Region/District 



City/State 



South (31) 

Atlanta Public Schools 
Austin Independent School District 
Charleston County School District 
Dade County Public Schools 
Dallas Independent School District 
DeKalb County School District 
East Baton Rouge Parish School District 
El Paso Independent School District 
Escambia County School District 
Fort Worth Independent School Dist. 
Fulton County Board of Education 
Garland Independent School District 
The School District of Greenville Co. 
Gwinnett County Public Schools 
Hillsborough County Public Schools 
Houston Independent School District 
Irving Independent School District 
Jefferson County Public Schools 
Metropolitian Nashville Public Schools 
Mobile County Public Schools 
Montgomery County PubUc Schools 
Orange County PubHc Schools 
Pinnnellas County Public Schools 
Polk County PubUc Schools 
Richardson Independent Sch. Dist 
Richmond City PubHc Schools 
Rockdale County School District 
Saint Lucie County Public Schools 
Tulsa City School District 
Wake County Public Schools 
York County Public Schools 



Atlanta, Georgia 
Austin, Texas 
Charleston, South Carohna 
Miami, Florida 
Dallas, Texas 
Decatur, Georgia 
Baton Rouge, Louisiana 
EI Paso, Texas 
Pensacola, Florida 
Fort Worth,Texas 
Atlanta, Georgia 
Garland, Texas 
Greenville, South Carolina 
Lawrenceville, Georgia 
Tampa, Florida 
Houston, Texas 
Irving, Texas 
Louisville,Kentucky 
Nashville, Tennessee 
Mobile, Alabama 
Rockville, Maryland 
Orlando, florida 
Largo, Florida 
Bartow, Florida 
Richardson, Texas 
Richmond, Virginia 
Conyers, Georgia 
Fort Pierce, Florida 
Tulsa, Oklahoma 
Raleigh, North CaroUna 
Yorktown, Virginia 
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Table O. Participating Public School Districts by Region (continued) 



Region/District 



City/State 



West (26) 

Albuquerque Public Schools 
Anchorage 

Cherry Creek Schools 

Clackamas County Educ. Serv. Dist. 

Clark County School District 

Colorado Springs School District 

Corvina- Valley Unified School District 

Denver Public Schools 

Eugene School District 

Fresno Unified School District 

Highline School District 

Long Beach Unifird School District 

Los Angeles Unified School District 

Mesa Public Schools 

Mount Diablo Unified School District 

Multnomah Educ. Service District 

Newport-Mesa Unified School District 

Portland School District 

Pueblo School District 

Rowland Unified School District 

Sacramento City Unified School District 

Salt Lake City 

San Juan Unified School District 
Tacoma Public Schools 
Washoe County School District 
Yakima School District 
International (1) 

Department of Defense Dependents Schools (DODDS) 



Albuquerque, New Mexico 
Anchorage, Alaska 
Englewood, Colorado 
Merylhurst, Oregon 
Las Vegas, Nevada 
Colorado Springs, Colorado 
Covina, California 
Denver, Colorado 
Eugene, Oregon 
Fresno, California 
Seattle, Washington 
Long Beach, California 
Los Angeles, California 
Mesa, Arizona 
Concord, California 
Portland, Oregon 
Costa Mesa, California 
Portland, Oregon 
Pueblo, Colorado 
Rowland Heights, California 
Sacramento, California 
Salt Lake City, Utah 
Carmichael, California 
Tacoma, Washington 
Reno, Nevada 
Yakima, Washington 

Alexandria, Virginia 



Note. N = 91 pubhc school districts. 
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APPENDIX B 



ArDerican Educational 
Research Association 



Opportunity to L^arn QuMtiosma5rA 



Umma Pot it ion Title: 

Bchool OUtrict: — 



Address i 



Telephone Nusiber: 



(Area Code) 



1. Please ch«ck c7) the grade levels of students tested for the school 
district t*stin^ program, 

Pr«-K X 1 2 3 ^ S 6 7 8 9 

10 11 12 

2. Does the district's testing program collect 

A. Norffl-referenced test data? Yes No 

b. Criterion-referenced test data? Tes No 

c. Perfoxisance-baaed test data? Y^b No 



If yes, please check the areas 



NRT CRT PBA 



Reading 

Mathematics 

Science 

Language (oral) 
Language (written) 
Other (specify): _ 
Other (specify): _ 



3. At what organirational levels are test data provided? (Check all 
±\ut apply) 



NRT CRT PBA 



a. District 

b« Hegion/Area/Zcne 

Bchool 

ClaasrooQ 
%. Students (special froupa) 
f. Students (individual) 



. 1 . e 



ERIC 



lit 



SI COPY AVAILABLE 



4. By what dwnographic characteristics are test datw provided? (Check 
all that apply) 

a. Raca/Ethnicity 

b. SES (or surrogate SES) 

c. Parents* lavtls of aducation 

d. Parents' Job status 

o. Other (specify) 

5. By what background information are test data provided? (Check all 
that apply) 

a. Teachers* years of teaching experience 

b. Teachers' attendance rate 

c. Students' attendance rate 

d. Other (specify) 

e. Other (specify) 

6. Do you have a district managed interval or segmented testing program? 
Yea No 



a. If yes, NRT format 

CRT format 

b. If no, is interval/segmented testing in your future plans? 

Yes 

No 

7* Does your office have the capacity to do cross tabulations of test 
data with demographic and background data? Yes . No 

8. Is there a district policy which would prohibit cross tabulations 

of test data with demographic and backgro\md datA? Yes No 

9. Do you use OIL as a crosstab in your data analysis? Yes No 

10. The following data elements have been used in the opportunity to 
learn definition: 

Exposure of students to the curriculum 
Time devoted to instruction 

Subject matter topics emphasized in cl asi^room 

Please list your suggestions to expand the definition of 
opportunity to learn (OTP, 
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2. 
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APPENDIX C 



OPPORTUNITY TO LEARN: THE NEED FOR 
A COMPREHENSIVE DEFINITION FOR SUBSTANTIVE DATA ANALYSIS 

Floraline I. Stevens 
AERA Senior Research Fellow 
National Center for Education Statistics 

Opportunity to learn (OTL) is an important macro variable to be considered when 
designing surveys, research and evaluations of education issues and for policymaking. 
Recent publications by Lapointe, Mead and Phillips (1989) and MuUis, Dorsey, Owen and 
Phillips (1991) report this information under the topic of opportunity to learn. The 
Lapointe, et al., study defined opportunity to learn as students' exposure to the material 
covered while the MuUis, et al., study described OTL as the amount of instructional time 
provided to students, and the subject matter topics emphasized in the classroom. 
Although both definitions are related, they are different. In addition, both definitions are 
limited to no more than two variables. Currently, there appears to be no comprehensive 
operational definition of opportunity to learn. 

This is a need to develop a comprehensive definition in order to use this macro variable 
to ascertain whether or not all participants in a study are in comparable learning 
situations before definitive outcomes/results are reported. Using opportunity to learn 
data, coupled with student outcome data provides insight and better understanding about 
why there are differences in the outcomes/results among various groups. Presently, 
outcomes are principally reported using demographic variables of gender, race and 
ethnicity, and parents, educational and income levels as the descriptors. Admittedly the 
two-variable disaggregation of outcome data is a step forward from global reporting of 
data; however, the inclusion of opportunity to learn data as a third or fourth variable 
would provide a more complete description of student outcomes or results. 

Discussions about opportunity to learn (OTL) have provided a much wider dimension to 
this variable and identified far more data elements to be considered. Some are familiar 
and would be viewed in a different context when investigating opportunity to learn. 
Although exposure to curriculum and subject matter and the devotion of time to the 
curriculum certainly contribute to a large part of the OTL definition, another important 
dimension to be considered is the quality of the delivery of instruction. Quality focuses on 
the providers of instruction, teachers, and school principals, not the students. For 
example, teacher attendance or absence has a marked effect on whether the subject 
matter is covered and effectively delivered to students. Quality of the delivery determines 
whether or not students master the curriculum. The services of substitute teachers versus 
the continuous presence of regular teachers will most certainly impact student 
achievement outcomes. Unfortunately, the substitute teacher factor is more prevalent in 
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urban, poor schools where thnre are large numbers if students who are at-risk. 

Further investigation of opportunity to learn involves the quality of the regular teacher 
who is responsible for the delivery of instruction, various levels of effectiveness of the 
delivery impacts whether students are able to master grade level curriculum or subject 
matter. Imbedded in the issue of quality can be teacher certification information, and 
whether or not inservice training has upgraded teachers, skills and strategies to effectively 
teach students from various social and cultural backgrounds. 

In a recent evaluation of a program planned for students in an inner urban school district 
(Alkin, Doby, and Lindheim, 1990), the reading scores did not increase. Traditionally, the 
outcome data would be reported by group (experimental vs. comparison) and by grade 
level. If expanded further, it would use such background variables as gender, race and 
ethnicity, and socioeconomic level. However, in this evaluation, case studies were 
conducted which brought forth information that tne reading program had changed to 
literature-based or core literature, and some teachers had difficulty using this approach. 
With this information, the opportunity to learn factor of quality of instructional delivery 
strongly impacted the student outcomes, e.g., reading scores. 

These few examples point out that opportunity to learn goes beyond student exposure to 
curriculum. A more comprehensive definition must also include those elements that 
impact the effective delivery of instruction. They may include: 

0 Interval test data covering segments of the curriculum 

0 Teacher attendance rates (substitute teacher days) 

0 Remedial/enrichment opportunities (tutoring, computer assisted instruction, 

monitored homework, etc.) 

0 Bilingual instruction for limited English-speaking students 

0 Coherent instructional delivery 

0 Principal leadership (an environment for learning) 

Transforming Qualitative Data into Quantitative Units 
for the Purpose of Data Analysis 

One of the main issues to be resolved with the opportunity to learn variable is how to 
ascertain the quality of instructional delivery provided to students in classrooms. In the 
most recent NAEP study by Mullis, et al. (1991), teacher certification and teacher 
self-reports of teaching activities and instructional emphasis attempted to address this 
issue as background information, and tangentially in the context of OTL, as a measurable 
variable. However, there is research on instructional processes and strategies that 
addresses what successful teachers do in their classrooms. Dole, Du^, Rochler and 
Pearson (1991) attempted to answer how reading comprehension instruction should be 
delivered. They did this by integrating research findings about comprehension processes 
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comprehension strategies, and teaching strategies to inform instructional practice. In a 
more general offering of suggestions about reading instructional delivery, Binkley (1986) 
offered information and suggestions about reading activities and lesson format. Another 
study by Dorr-Bremme, Keesling, and King (1984) presented research findings based on 3 
years of classroom observations regarding reading instructional delivery practices in 
elementary schools. As in reading research, there is the assumption that similar research 
findings on instructional delivery are available for mathematics,, science, language arts and 
other subject matter areas. 

The question is: How do you transform these and other research findings on instructional 
delivery proficiency into quantitative units? Teacher self-reports from guided questions, 
observational checklists, reviews of lesson plans, homework assignments and other related 
tasks/materials should be considered in devising a means to quantify what is thought of as 
qualitative data. Without qualitative information included in OTL, the analysis of student 
outcomes, particularly academic achievement, would be incomplete. 

Lapointe, et al., (1989) in their study transformed ratings of opportunity to learn into 
values of 1 and 0. Follow-up of the analysis procedures may prove to be helpful. 
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APPENDIX D 



OTL Survey Followup Interview Protocol 

Name .Telephone 

School District . Code_ 

Hello. I am Fioraline Stevens. I am calling to talk with you regarding the 
Opportunity tc Learn survey form that you completed and returned to me. 
First of all, thank you for your support of my research as an AERA fellow 
at the National Center for Education Statistics, U.S. Department of 
Education. 

This followup interview should only take a few minutes. I need to add 
some background information about your school district. Also, in response 
to your comments/ lack of comments about what should be included as 
elements in the opportunity to learn definition, I need some additional 
information. 

1. What percent of your students are Chapter 1 students? % 

2. What percent of your students are white? % 

3. How would you describe the academic performance of the majority of 
the students in school district in reading, mathematics, and written 
language. 

Above Average Average Slightly Below Below 

Reading 

Mathematics 

Written Language 

4.1 (A) On your survey form, you recommended that the following elements 
should be included to the definition of opportunity to learn: 

1. 

2. 

3. 

4. 

5. 

4.2 (B) Several of your research and testing colleagues recommended 
elements for inclusion to the definition of opportunity to learn. 
(Read list of elements.) 
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5. Would you give your rationale or comments about why any these 
elements should or should not be included in the opportunity to learn 
definition. 



6. Do you think that opportunity to learn variables should be used in the 
analysis of student outcome data, e.g., test scores? Yes No Why? 



7. Now that you have had time to think about opportunity to learn, are 
there additional elements you would like to add to the definition? Why? 



8. Any other comments or questions regarding this research would be 
welcome at this time. 



Thank you for your assistance. 
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